DOCUSENT RESUME 


_ BR 139 459 . 4 7 <3c¢ 770 279 
AOTHOR 7. j Alfred, Richard L. : . a oF _ 
.: TITLE.” “Measuring the Outcomes of Higher Edacation; Two ‘ 


Approaches. Based on Program Charact2ristics and 


_ 3 ‘ 4 ; os Perceived Needs of External Agencies. 
‘rPUB DARE ‘Now 76 
NOTE ‘ % _ 3p. an ‘ a Ris ° 
EDRS PRICE, > MF=$0.83 HO-$1.67 “niha. "Postale. ; 


v DESCRIPTORS \ Dectsion making; *£ducational Accountability; . 
se . » “Informatian Needs; Juniar Colleges; *Hedsurement 
Pe. tn - Techniques; Models; *#P25t Secondary Education; 
: : Productivity; Program Costs; .*Prograa aed 
Resource Allocations; State aids Siig SULUOES 
, ’ ' Education ‘ 
IDENTIPIERS Outcome, Heasures;’ *0utzomes of Educ ation’. , : 
i bd ae ’ ’ . ‘ aN 
ABSTRACT - : Re 
Tne developasnt and us2 des prograa oatcomes data can’ 
. be an important,. positive institutional asset as ext2radal highar : 
edaeztion agencies more frequently teqiire, institutions to proviije 
\ evidence of outcomes in reghurn for resources. Argning that more and , 
..better research needs £0 be done on. the, outcome’ of: community. college + 
\edacation, ‘this document utilizes two follow-up. studies of stadents 
‘who grafluated from career prograas to illustrate how ‘outcomes' data ox 
may b2 organized to relate to ‘career programs and thir’ 
characteristics, and to the information needs of external agenci2s. A 
- ¢two-disensional outcomes model gs: developei, with the first Aiménsion 
’ coaprised of three categories--career preparation, transfer ‘ 
Preparation, and public service--and the second coaprised of three. 
“ time dimension categories-7+before, during, and after college. This 
taxonomy permits simultaneous classification of each program outcome ‘ 
: oe type of function involved and its't2nporal sequence in the colleg? 
" nvitonment. The resulting organization of information. proviies . 
re cEeSReaDy. answers to questions asked i the . ‘ 


A ~ 
C7 


: resourre- allocatiop decision-making process. (JDS), 


. Ss a : 
‘ a 
> ’ « . P s 
* . s . ws ‘ 
t a eTiTrrrr TT errrerrrr errr Teer ress TOO REE EEE EERE RE EEE REE EE 
* Documents acquired by ‘ERIC include many informal. unpublished * 
'* gaterials%ot available from other sources. ERIC makes every effort * 
* to obtain the best copy avail le.’ Nevertheless, items of marginal. 
: * reproducibility are often encdunterei and this affects the quality + 
“A: * of the microfiche and hardcopy projuctions ERIC Mikes available- «¢ 
oN * via the ERIC Doqument‘ Reproduction Service (EDRS)*%. EDRS is not * 
. ,  ™® Fasponsible for-the quality. of the ‘original document. Reproductions * 
* suppliei by EDRS are the best that can be made from the original. * 
. s * 


PROEREE SE REE EERE EE EEERE HE HEHE PHA RAEE EEE EEREEEERERE RES REEL ESSER EEE S 


- 


. 


. Measuring. the Outcomes of Higher Education 


"Two Approaches BaSed on Program 
Characteristics and Perceived Needs 
of External Agencies 


US DEPARTMENT OF HEALTH, 
EOUCATION 6 WEL FAR 
MATIONAL INSTITUTE OF 

EOUCATION 


“-ToIS, DOCUMENT WAS BEEN Re&Pac 


OuCEO EXACTLY AS RECEIVED ROS 
THE PERSON OR ORGANIZATION ORIGIN 
ATING IT POINTS OF WEW OR OPINION 
STATED OO NOT NECRSSAR LW REPRE 
SENT OFEICIAL NATIONAL INSTITUTE O 
EQUCATION POSITION OR POLICY 

‘ 


« J ¢ . 
. — Richard L. “Alfred 


Director, 


Office 


of Planning, Research 


\ *" eCollege 


November, 1976 


\ 


_ and Development 
New York City Community 


\ Researchers and planners have nown for a long time that 
os ; 


eveluaton is supposed to begin with a statement of educational 


|, Wieetives. And every good manager knows that evaluation in- 


cludes, an assessment of program costs which, is Supposed to in- 


% 


volve some consideration of program outcomes. In the field of 


community college education, however, these basic rules have 


often been ignored by administrators and external agencies 


attempting to carry out evaluation studies., Educational master 


plans; for example, developed by. gommunity colleges, have little 


organizational structure. 


ie 


Two-year. college faculty, department chairpersons, and 


-to say abQue outcomes or costs even iced these plans. contain 


La variety of propositions concerning Financing, programs and 


deans engaged in, making annual recommendations on.how academic * 


departments should be budgeted, as a matter of -policy do not 


include any consideration of outcomes. Year-in and year-out, 


program budgets are determined Ona percentage basis as an in- 


grease or decrease from the previous year's budget. This practice, 


is long as it is continued, can~ hardly be expected’ to encourage 


- 
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faculty to deal with outcomes in their programs and budget. 
++ decisions. © 


. % A External Pressures’ for Accountability 
While is is ‘not Gnreecly ‘clear why the outcomes issue has 


been avoided by colleges ‘and. their internal constitutuencies, it 


° 


is becoming snoreasing!y clear that state boards and legislatures 


é * 


are interested in higher “education outcomes. Policy makers @re 
\s » becoming gidavcsienked rather than means-oriented. The attitudes 
"'s “of funding sources clearly reinforce this trend: colleges should 
“be Yewarded not only for the gumber of programs and services they 
* offer, but for the_ outcomes they produce and the cost Banadities 


ae 


to the consumer. 
: iN % 


\ > . x % - 
‘Although it is perhaps unfortunate that higher education 


agencies are requiring institutions: to provide evidence of out- 

comes in. return Pe, een it is likely that many policy: 
, questions ‘would be much easier to settle if we had a better 
understanding. ef ‘the outcomes Pee The issue of program 
budgeting, for ene, is one’ topic ‘that has stimulated’ a good 
‘deal -of debate: At what level should programs be funded? what 
types of measures should be used to determine program funding? Ss 
Who should make the’hard decisions on funding? ‘What criteria 
should be used to determine funding levels? Should programs be 
_funded on a "relative" basis (as an increase or decrease over 
the previous year's budget) or on an "absolute" basis as a measure 
‘of the minimum amount of resources needed? Such°issues would be 


readily resolved if data were available on the outcomes that are 
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* actually generated by two-year college programs. How many stu- 


i 


dents are enrolled in a program and what number of credit hours 


do they represent? How many complete their degree requirements 


_ within’ two years, three years, and so forth? What is the cost 


per FTE student? Do the outcomes produced meet or fall short of 


a minimum -acceptable standard? What are the cost benefits to 


the consumer? 
The major premise of this presentation is that much more ~ : 


research needs to be done on the outcomes of community college _ 


‘education before community colleges can be truly evaluated or 
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receive their, fair share of resources. The studies besextivad 
herein relate to outcomes in career programs,to the chasbtens 
istics of these programs,and the information needs of external 
agencies. Suggestions are mada. as to how outcome Measures might 
be used to improve the position of higher education institutions a 


in their quest for resources. 


‘ 


,, Outcome Measures and Decision Making 
While there are many possible methodg that can be used to 
measure outcomes in community college programs, a fundamental pur- 
pose of setemmmnctented research should be to produce information 
that can be used by decision makers to determine the resources for 
career - programs. Outcome data are most likely to be useful if 
they are based on an understanding of the resource allocation 
process itself, particularly that involved in the velationship 


between the institution and- the state. 
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The need for rendering a decision relative to the resources 


for community college programs implies the existence of two 


“fundam ntal conditions: some recognized educational. dbjective 


| < e . 
. and limited resources for achieving this objective. Decisions 


on mages education appropriations typically involve a choice. 


between available means by which manpower and educational re- 


snitconianee cahi be met. In a densely ‘populated urban region, asc 


example, these means might include funding two~yéar college pro- 


: ‘ = : 
grams to meet the technical and manpower requirements of business 
and industry; support for’ baccalaureate degree-granting insbitu- 


tions to meet mid-management and human service needs; and support 


“for professional degree-granting institutions to provide trained 


professionals for emerging manpower needs in health, engineering, 
° : 
and the social sails lade 3 
Every Appropriations decision is predicated on a belief in 


the existence of a causal Helavioauhio between some educational 


‘ outcome and the resources allocated to aah gue that outcome. 


Rational decisions concerning program resources can be rendered 
by consulting the available information in’a college tegarding 


the outcomes it is trying to produce and balancing thig\ with in- 
- : \ 


formation about the outcomes it has actually produced. The 


principal function of outcome research is to extend this fund 


of information to help decision-makers--both within and outside 


of the fnetitution--to better understand the consequences of the 


resources they are employing. 
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Two studies were conducted on outcomes in curriculum pro- 


‘ 


grace in the Allied (Geakee and Natural Science and Engineering 


Technology divisions of New York City Community College during 


* 


a ? the Fall of 1975. These sitwdlises were designed to identify stu- 


‘4 dent outcomes pertaining to their career patterns, their transfer 
#4 plans, their perceptions of acollege Aiemibenite. and their 
. attitudes toward work and further OuER ETOH The study populations 
P *, 3 Lonuleted ‘of 922 graduates of the Siviision of Technology between . f 


1969 and +1975, and 595 respondents Seon iis Divis sion of Allied 


Health and Natural Sciences. Questionnaires were sent to geate 
" : uates of these programs’ during the Spring and Fall of 1975. 
peikoniias tale 44 percent returned veable questionnaires, The 're- 
sponse rates varied over the seven-year period ranging from a.’ 
| ", Low. of 35 percent in 1969 to A high of 60 percent in 1975. The 
i: . “overall response rate of 44 ‘percent was considered a gratifying 
7 return for studies of this kind. ® — — 
Because the number of outcome measures, used in each study 
was very large, a taxonomy was developed not only for elaaaktys e 
ing existing measures but also for suggesting add viene ones 
(see Figure I), This taxonomy was based on the traditional fun- ‘an 
_ etions of two-year colleges: teaching, student development, and, 
Bubiie service. The first dimengion of the taxonomy was conpritzed 
I a of three ithewaedace career preparation, transfer ‘preparation, | 
| and public service. The second diménsion enuclon the time dimension 


in which outcomes in each category were measured: before college, ° 


‘ during college, and after college.’ 
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“Taxonomy of Outcome Measures 


Transfer Preparation 


pre-college aptitude in . 
reading ,*math and 
writing (interest tests) 

high’ school curriculum 


high school grades, * 


. 


grades (general courses) 
grades (major field) 
change of major 
withdrawal 
graduation/honors 
Student perceptions 
choice of transfer 
institution 
time ‘to finish program 


& ES 


; enrollment in trangfer 

institutions — pad 
grades a 
advanéed degrees , = “ 


¥ 


honor& , : 
tudent- perceptions. / 


. years to finish Paget 


‘degrees 


‘Public Service » 


community: activitieg 
while in i schoo 


community activities 
_ while in college 


4 
_- 


“place of residence: \ 


fm (in or gut of 
“ community) 
govmunity APEEGS TARE * 


a 
a F - 
: 


the categories in each dimension are to some degree Sriter 
dependent. Certain transfer and career preparation outcomes, for 
example, can be evaluated through analysis of the same outcome 
_ measures. Thus, one of the critical outcome Measures of a student's 
preparation for transfer or a career is his asada point average 
in ‘the major field:'of study. By the same token, one of the most 
improtant aspects of a student's public service is his partici- . 
pation in organized diisaeigies activities. Public service can be 
assessed in terms of the quality and quantity of nee involve- 


ment in the community at every point in his relationship with the 


; college--before he enrolls, during enrollment, and after graduation, 2 
Each outsome is classified simultaneously by the type of function dace 
involved and its tempogal sequence in the .college envicotwent. ae 
parptiurieaa Functions. The ‘institutional functiok di- 
mension was used to assess the effects of college programs on e. 
various student outcomes under consideration. The outcomes in 
, each study differed according to the characteristics of the pro- 
‘grams being studied: and the function being examined. For example; 
students enrolled in the Allied Health program are required to 
successfully complete licensure examination kefore they can ? 
practice in their career field. The outcomes investigated *in this 
study were those under the general rubric of career _preparation 
‘but particular to the exaninanten results and curricular per- 
ceptions of Students (se& Figure 2). Students enrolled in Tech- 
nology programs, on his othe hand, are not required to take a 


certification exam. as a condition for career entry. The focus 
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a ee & 2 ; . . 
. ; if a a ene 
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: ‘shifted in this study, from a concern with axamingtson outcomes 
fi . ie 
Fi ‘ a -ooncern with job performance. Measures such as ‘salary, a 
4 


mobility, supervisory responsibility and employer sede ong 


¢.. % were used to evaluate performance on the job (see Figure 2) ~ 
i> poe : PA . 
N! 2 _ The Time Dimension. Classified into a temporal dimension, 


the functions in Figyre 2 portray the sequence of the outcome 

Gosazoan anid in both studies. Although it is not often cons 

i 4 sidered in the deliberations of educational policy makers, time 

; : “ig a critical element. Is it ‘more appropriate to ee decisions : ‘ 

ae on the basis of immediate outcomes of the’ sash age experience-~ | 

| that ty cheee that are evident after ney a brief span of time-- 
or the anaes which show the long term effects of: higher edu-. 

“ eation? This is a question that ‘college officials’ and external ° 

iv 9 bie ier al must grapple with. ‘From the _standpoint of those de- | e: a 

términing institutional support, the long-term effects are too 

remote and too difficult to comprehend to be ‘used in appropri- 

| —— ations denbelonas chal primary interest lies in nach more immedi-_ 

es ie ‘até outcomes: How .many students were graduated? How many retaiied : 


_ jobs in the local cine ene much was spent to cee arenes 
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served by the results of the studies reported herein? aes who 
wish to hold institutions accountable should become the. “recipients 


of outcomes data and ar use such data to examine results 


and costs: in the educational enterprise. The exchange of infor- 


a . i 
mation between institutions and agencies cannot. but help-to .bring- 
some rational element to all questions relating to resources. 
Published results of outcomes studies lend a focus to the 


. 


types of resources needed to operate programs. They separate 
questions related to puqeeed and procedures and permit educa 
/t10nal polacy makers to develop.a better understanding of the 
educational process. In this sense, outcome data is a weapon. 

It can be used to educate the policy makers about the-academic 
and institutional facts of life, providing them a far better, 
more euitiacn and comprehensive picture of the academic en-> : 

‘ ; 

terprise than they now have. Tending to relationships with state 
agencies has become an increasingly important and time consuming 
task for community colleges. Failure to provide the types of ine 


formation that these agencies should use in the resource allo- 


cation process could result in some severe short-term. penalties. 
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